Record whether a command's last run failed (fixes #187) - #197
Conversation
A command that exited nonzero was not remembered. If its declared outputs existed when the next build started, that build reported "Nothing to do (up to date)" and exited 0, so the failure disappeared: build 2 (-k): Build completed: 0 commands (1 failed) exit 1 build 3 (-k): Nothing to do (up to date). exit 0 The bisect on the issue localises it to 6ae36d1, which made the index write unconditional and justified it in the commit message: Failed outputs don't exist on disk, so stat fails and they're detected as changed. That holds only when the failed command produced nothing. `cp %f %o && false` writes its output and then fails, so the stat succeeds, the hash matches what the index just recorded, and there is nothing left to notice. Before that commit the guard was `stats.failed_jobs == 0 && !opts.dry_run` -- exit status was honoured bluntly, by not writing the index at all, and that commit removed the guard while arguing about a different category. So exit status becomes what it should have been: a recorded field. CommandEntry carries `failed`, set from the scheduler's job results, and a command whose previous entry is marked failed re-runs regardless of its signature. Restoring the old guard was the alternative and is worse -- it also discards the successful commands' state, which is the thing 6ae36d1 legitimately fixed. Routing matters here and is not interchangeable. The re-run is pushed through `changed_outputs`, not `forced_cmds`: forced_cmds are OR'd into the affected set at cmd_build.cpp:2058 *after* collect_affected_commands has computed its closure over changed files, so a forced id adds exactly one command and its consumers do not propagate. A failed command that half-wrote its output must reschedule whatever read that output. INDEX_VERSION 19 -> 20: RawCommandEntry carries a flags word, 80 -> 88 bytes. Two e2e scenarios, both observed failing first. The second is the one that pins the routing: a producer writes PARTIAL and fails, its consumer copies that to the final artifact, and after the cause of the failure is removed -- with nothing else changing -- the producer must re-run and the consumer must pick up the corrected output. Before this change that assertion read "PARTIAL", with the build reporting success. Also verified by hand: the failure survives repeated builds (3 and 4 both exit 1), and clears on the first successful run, after which the next build is a no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
PR metricsPerformance (gcc example, Linux)
Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1). Internal statistics (gcc example, up-to-date dry run)
Counters from Binary size (Linux)
Code churn (whole codebase, last 30d)
Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 3585 lines were deleted in the window in total, most of them older than it. Where the churn is
Test coverage (lines)
103 files · 15185/17508 lines covered Deltas vs main@a1277e1d4. Updated for 7050628 |
From an adversarial review of this PR. The field said "failed during the run that wrote this index" when it must say "has not succeeded since it last failed", and that difference reopened #187 through two doors, both reproduced: putup -B . -k # a/ fails, recorded putup -B . -k outb.txt # target build; a/ does not run putup -B . -k # Nothing to do (up to date). exit 0 and the same via a scoped build, where merge_out_of_scope_commands copied the old entry forward without its failed flag. tup remembers in both cases; this forgot. The flag is now seeded from the previous index at detection, carried through any build in which the command does not run, and cleared only by a successful run of that command. merge_out_of_scope_commands carries it too, unconditionally: an out-of-scope command did not run, so its recorded state cannot have advanced. The consumer scenario did not pin what the commit message claimed it pinned. Routing retries through forced_cmds -- the alternative the message argues against -- left both scenarios passing, because final.txt had never been created and the consumer was scheduled by missing-output detection whatever the routing did. It now builds successfully first so the consumer's output exists holding V1, and asserts it reaches V2. Verified by mutation: that routing change now fails the test. Two scenarios added for the doors above, and the flags/reserved comments corrected -- there is no CommandFlags type, and Hash256 is std::array<std::byte, 32> so nothing needed 8-byte alignment; reserved makes the tail padding explicit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
A command that exited nonzero was not remembered. If its declared outputs existed when the next build started, that build reported "Nothing to do (up to date)" and exited 0 — the failure simply disappeared:
Why it was wrong, from the bisect
The bisect on the issue localises this to
6ae36d1c2, which made the index write unconditional and justified it in the commit message:True only when the failed command produced nothing.
cp %f %o && falsewrites its output and then fails, so the stat succeeds, the hash matches what the index just recorded, and there is nothing left to notice.The guard it replaced was
stats.failed_jobs == 0 && !opts.dry_run— exit status was honoured, bluntly, by not writing the index at all. That commit removed it while arguing about a different category of recorded state. This is the shape #189 describes: with the categories scattered, a plausible local argument can remove a leg rather than merely forget one.The fix
Exit status becomes what it should have been — a recorded field.
CommandEntrycarriesfailed, set from the scheduler's job results, and a command whose previous entry is marked failed re-runs regardless of its signature.Restoring the old guard was the alternative and is worse: it also discards the successful commands' state, which is the thing
6ae36d1c2legitimately fixed.INDEX_VERSION19 → 20;RawCommandEntrygains a flags word, 80 → 88 bytes.Routing is not interchangeable here
The re-run is pushed through
changed_outputs, notforced_cmds.forced_cmdsare OR'd into the affected set atcmd_build.cpp:2058aftercollect_affected_commandshas computed its closure over changed files, so a forced id adds exactly one command and its consumers do not propagate. A failed command that half-wrote its output has to reschedule whatever read that output.This was flagged on #189 as one of two constraints the bisect explicitly did not measure, so it is verified here by test rather than assumed.
Tests
Two e2e scenarios, both observed failing first. The second pins the routing: a producer writes
PARTIALand fails, its consumer copies that into the final artifact, and once the cause of the failure is removed — with nothing else changing — the producer must re-run and the consumer must pick up the corrected output.Before this change that assertion read:
Also verified by hand: the failure survives repeated builds (3 and 4 both exit 1), and clears on the first successful run, after which the next build is a no-op — so this does not trade a stuck-green for a stuck-red.
Verification
make test: 142417 assertions, 639 cases, 32 e2e shards.make tidy,make iwyuclean.Noted, not fixed
The new verbose line reads
Failed last run:with an empty name, becauseDisplayis only set from^ ^markers. That affects every-vrouting diagnostic, not just this one, and is worth its own change.🤖 Generated with Claude Code
https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA